Feature/evals dod#175
Conversation
…pace and restructuring yaml configuration
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a rubric-based LLM-as-a-judge evaluation system for agent skills, adding documentation, ignore rules, and a test suite for the definition-of-done skill. Feedback highlights that the new .gitignore rules for evals directories are redundant or ineffective due to Git's directory traversal behavior. Additionally, the evaluation prompt path in evals.json should be updated to use the correct relative path from the repository root to prevent execution errors.
| # Keep evals inside un-ignored skill folders | ||
| !**/evals/ | ||
| !**/evals/** | ||
|
|
There was a problem hiding this comment.
In Git, if a parent directory is ignored (due to the * rule at the top), Git will not traverse into it. Therefore, !**/evals/ and !**/evals/** will not work for any third-party/ignored skill folders because their parent directories are excluded.
For the explicitly unignored skill folders (like !definition-of-done/), all of their contents (including evals/) are already unignored by default, making these rules redundant for them.
Since these rules are either redundant or ineffective, they should be removed to keep the .gitignore clean.
…evals.json Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2 participants
Add proof of concept evals to definition-of-done.
#174
Follow up work coming for a better way to ignore directories because that introduced the pattern of rules based custom arguments.
Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.